Papers with policy model
Reflect-RL: Two-Player Online RL Fine-Tuning for LMs (2024.acl-long)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. |
| Approach: | They propose a two-player system to fine-tune an LM using SFT and online RL . they use negative example generation to enhance error-correction ability of the reflection model . |
| Outcome: | The proposed system outperforms SFT and online RL without reflection on a GPT-2 XL 1.56B model. |
Large Language Model-based Human-Agent Collaboration for Complex Task Solving (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to the development of LLM-based autonomous agents. |
| Approach: | They propose a Reinforcement Learning-based Human-Agent Collaboration method which trains a policy model to determine the most opportune stages for human intervention within the task-solving process. |
| Outcome: | The proposed method improves human-agent collaboration significantly through well-planned, limited human intervention. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Modeling Multi-Action Policy for Task-Oriented Dialogues (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to learn dialogue management only predict one action per turn, limiting expressive power of the conversational agent and introducing unnecessary turns of interactions. |
| Approach: | They propose a model based on a recurrent cell called gated Continue-Act-Slots that overcomes the limitations of existing models and proposes a novel policy model that predicts multiple acts for each turn. |
| Outcome: | The proposed model outperforms existing models on the task of predicting multiple acts for each turn. |
Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance medical reasoning lack high-quality data. |
| Approach: | They propose a medical knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework that uses rare disease knowledge to synthesize distribution-controllable reasoning questions. |
| Outcome: | The proposed method outperforms existing methods across ten medical benchmarks and achieves up to 5.93% gain on rare diseases tasks. |
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective (2025.acl-long)
Copied to clipboard
| Challenge: | if rewards are imperfect, they can adversely affect the alignment of large language models (LLMs). |
| Approach: | They propose a bias-agnostic method to address the issue of reward unfairness from a resource allocation perspective without specifically designing for each type of bias . they apply methods Fairness Regularization and Fairness Coefficient to achieve fairness in rewards. |
| Outcome: | The proposed method achieves fairness in rewards while minimizing biases . it can be applied to verification and reinforcement learning scenarios . |
Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies attempt to obtain optimal or suboptimal arrangements based on statistical results or using dataset-based search, but these methods increase inference overhead while leaving the model’s inherent order bias unresolved. |
| Approach: | They propose Dual Group Advantage Optimization (DGAO) which aims to improve model accuracy and order stability simultaneously. |
| Outcome: | The proposed method improves model accuracy and order stability while penalizing order-sensitive or incorrect responses. |
Aligning Large Language Models with Implicit Preferences from User-Generated Content (2025.acl-long)
Copied to clipboard
Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, Meng Jiang
| Challenge: | Existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale. |
| Approach: | They propose a framework that leverages implicit preferences in unlabeled user-generated content to generate preference data. |
| Outcome: | The proposed framework transforms user-generated content into user queries and generates responses from the policy model. |
Boosting Policy and Process Reward Models with Monte Carlo Tree Search in Open-Domain QA (2025.findings-acl)
Copied to clipboard
Chi-Min Chan, Chunpu Xu, Junqi Zhu, Jiaming Ji, Donghai Hong, Pengcheng Wen, Chunyang Jiang, Zhen Ye, Yaodong Yang, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Experimental results show that our approach can effectively improve the performance of both the policy model and the reward model. |
| Approach: | They propose to use Monte Carlo Tree Search for both policy model improvement and reward model improvement to bridge it to more subtle open-domain question answering. |
| Outcome: | The proposed approach surpasses existing methods for annotation and training data with fewer data points and achieves better performance in test-time scaling strategies. |
Towards A Better Initial Policy Model For Scalable Long-CoT Reinforcement Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Long-CoT reasoning and reinforcement learning are demonstrating remarkable performance and scalability, however, there is a lack of systematic guidelines for obtaining a better initial policy model. |
| Approach: | They propose a systematic guideline and a novel Re-RFT method to obtain more efficient reasoning patterns from different initial models. |
| Outcome: | The proposed method surpasses DeepSeek-R1-Distill-Qwen-14B model by 4.6%, demonstrating its effectiveness and superiority. |
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to align Large Language Models with human preferences fail to maintain general knowledge and alignment when faced with personalized preferences. |
| Approach: | They propose a method that utilizes the initial responses of the reference model to mitigate forgetting while accommodating personalized alignment. |
| Outcome: | The proposed approach mitigates forgetting while accommodating personalized alignment while preserving global knowledge and general alignment. |
Offline Reinforcement Learning for LLM Multi-step Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly applied to complex tasks requiring multi-step reasoning. |
| Approach: | They propose an offline method for enhancing multi-step reasoning by optimizing the soft Bellman Equation by combining a policy model and a value function. |
| Outcome: | The proposed method surpasses existing methods on multi-step reasoning benchmarks and can be extended to multi-iteration frameworks when additional resources are available. |
Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
Hao Xiang, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Ben He, Le Sun, Jingren Zhou, Junyang Lin
| Challenge: | Prior research focused on developing data generation methods, while insufficient attention has been paid to quality control mechanisms and often produces inaccurate and unhelpful data. |
| Approach: | They propose an algorithm that automatically generates high-quality preference data, eliminating manual annotation requirements. |
| Outcome: | The proposed algorithm outperforms baselines in human preference alignment and reward optimization. |
Enhancing Reinforcement Learning with Dense Rewards from Language Model Critic (2024.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences, but the sparsity of these signals can lead to inefficient and unstable learning. |
| Approach: | They propose a framework that utilizes the critique capability of Large Language Models to produce intermediate-step rewards during RL training. |
| Outcome: | The proposed framework improves sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation. |
TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Mainstream approaches to aligning large language models heavily rely on human preference data. |
| Approach: | They propose a framework that fine-tunes a policy model using pairwise feedback data automatically mined from its outputs. |
| Outcome: | The proposed framework outperforms the base model with an average win rate of 69.7% across seven conversational or instruction-following datasets. |
Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reinforcement learning systems lack verifiable reward mechanisms for long-form question answering . current systems lack reliable long-term answers due to lack of factual content . |
| Approach: | They propose a framework for reinforced verifiable informativeness optimization . it defines informativeness as measurable and externally verifier objective for RL . |
| Outcome: | Experiments show that RioRAG achieves higher factual recall and faithfulness . the proposed framework is based on a framework that uses nugget-centric verification with cross-source checks . |
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models generate unintended outputs due to their unsupervised nature. |
| Approach: | They propose a method to construct preference pairs of selected and rejected LLMs by repeated random sampling to improve alignment performance. |
| Outcome: | The proposed method improves performance as the sample size increases. |
Mutual-Taught for Co-adapting Policy and Reward Models (2025.acl-long)
Copied to clipboard
Tianyuan Shi, Canbin Huang, Fanqi Wan, Longguang Zhong, Ziyi Yang, Weizhou Shen, Xiaojun Quan, Ming Yan
| Challenge: | Experimental results show that this iterative approach leads to consistent improvements in both the policy model and reward model. |
| Approach: | They propose a method that iteratively improves both the policy model and reward model without requiring additional human annotation. |
| Outcome: | The proposed method improves both the policy model and reward model without human annotation. |
Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent self-training approaches have reduced reliance on human-labeled data, which limits their scalability. |
| Approach: | They propose a team-based self-play algorithm that iteratively refines alignment without additional human supervision. |
| Outcome: | The proposed algorithm outperforms baselines and LLM benchmarks in the self-supervised setting. |
AgentRM: Enhancing Agent Generalization with Reward Modeling (2025.acl-long)
Copied to clipboard
Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing LLM-based agents have strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. |
| Approach: | They propose a reward-based generalizable reward model to guide the policy model for effective test-time search. |
| Outcome: | The proposed agentRM outperforms existing agents on held-in tasks by 8.8 points on average. |
Self-Training Large Language and Vision Assistant for Medical Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for collecting medical data are expensive and time-consuming. |
| Approach: | They propose a method to train a large-scale LVLM capable of auto-generating medical visual instruction data to improve data efficiency. |
| Outcome: | The proposed method shows that it performs well across three major visual question answering (VQA) benchmarks. |
MemPO: Self-Memory Policy Optimization for Long-Horizon Agents (2026.findings-acl)
Copied to clipboard
Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo, Yongbin Li, Jinli Suo
| Challenge: | Existing methods for long-horizon agents introduce the external memory module and look up the relevant information from the stored memory, which prevents the model from proactively managing its memory content and aligning with the agent’s overarching task objectives. |
| Approach: | They propose an algorithm which enables agents to autonomously manage their memory during interaction with environment and selectively retain crucial information. |
| Outcome: | Extensive experiments show that the proposed algorithm achieves absolute F1 score gains of 25.98 over the base model and 7.1 over the previous SOTA baseline while preserving task performance. |
Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing the performance of large language models require expensive manual annotations. |
| Approach: | They propose an offline direct preference optimization method that collects preference pairs through iterative sampling and execution feedback to improve model confidence. |
| Outcome: | The proposed method improves performance on three reasoning tasks and shows a 3.6% improvement over the standard method. |
Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification (2026.findings-acl)
Copied to clipboard
Yuxuan Wan, Tianqing Fang, Zaitang LI, Yintong Huo, Wenxuan Wang, Haitao Mi, Dong Yu, Michael R. Lyu
| Challenge: | Recent advances in Deep Research Agents (DRAs) are transforming automated knowledge discovery and problem-solving. |
| Approach: | They propose an inference-time scaling of verification wherein an agent self-improves at test time by evaluating its generated answers. |
| Outcome: | The proposed model outperforms vanilla agent-as-judge and LLM judge baselines by 12%–48% in meta-evaluation F1 score. |
Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)
Copied to clipboard
Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
| Challenge: | Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity. |
| Approach: | They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states. |
| Outcome: | The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters. |
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance. |
| Approach: | They propose a method that reweights rewards by a factor approximately proportional to Evidence Gain and assigns higher weights to high-quality traces without requiring costly computation. |
| Outcome: | Experiments on mathematical reasoning benchmarks show that Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally. |
Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Chain-of-Thought (CoT) methods struggle with consistency and verification in complex reasoning tasks. |
| Approach: | They propose a framework that integrates structured knowledge representation with learned planning. |
| Outcome: | The proposed framework outperforms existing Chain-of-Thought (CoT) methods on math reasoning, logical reasoning, and coding tasks. |
Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning tasks and general tasks. |
| Approach: | They propose a "Verifier-free Intrinsic Gradient-Norm Reward" that uses only the policy model itself. |
| Outcome: | The proposed reward outperforms the state-of-the-art RLIF baseline INTUITOR on math benchmarks and shows cross-domain transfer to code benchmarks when trained only on math data. |
Verified Critical Step Optimization for LLM Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Critical Step Optimization (CSO) focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Approach: | They propose a method which focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Outcome: | The proposed method outperforms the existing methods on GAIA-Text-103 and XBench-DeepSearch while requiring supervision at only 16% of trajectory steps. |
Joint Optimization of Training Data and Policy in RLHF (2026.findings-acl)
Copied to clipboard
Zhuohao Yu, Jiali Zeng, Weizheng Gu, Mengyuan Sun, Yidong Wang, Fandong Meng, Jie Zhou, Shikun Zhang, Wei Ye
| Challenge: | JODP optimizes policies on fixed training inputs, limiting the diversity of learning signals. |
| Approach: | They propose a framework where policy generates improved variants of training problems to enhance its own learning. |
| Outcome: | The proposed framework improves on safety alignment tasks by allowing 4B models to reach 8B model performance with less than 1% additional computational overhead. |